tests+docs(design): the period-honorific rows declare tolerated; W2 keeps its normative half - #505
Merged
Merged
Conversation
…6/7 refuse ASCII punctuation The 2026-09-01 CJK demotion swept the case table for a comma or an ASCII letter. A trailing ASCII period is neither, so three fix(#320) probe rows -- '田中さん 様.', '김민준 씨.' and '김민준 양.', each the spaced twin of a comma row that WAS demoted -- stayed on the contract tier. A period standing after a CJK honorific is a listing artifact no writing system produces, which is exactly the class that demotion names; the rows carry tolerated=True now. The criterion follows the doctrine rather than sampling it: a shape 6/7 tag admits no ASCII character but the space between two name words, so a period, a digit and a bracket are refused beside the comma and the Latin letter already refused. That gate is a property of the shape TAG, not of CJK text -- an untagged row may carry ASCII, and the four mixed-script rows d562164 reviewed by name keep the contract tier untouched. Measured 2026-09-05 by a wider sweep (CJK-bearing untolerated rows carrying any non-space ASCII, grouped by kind): eight texts before this commit, five after -- the period class gone, the letter class whole, and '山田 太郎 (マイケル・ジャクソン)' deliberately kept, its ASCII parentheses being one spelling of a delimiter the writing system does produce. corpus_cjk.jsonl 73 -> 70 and corpus_cjk_tolerated.jsonl 26 -> 29, the identity holding at 99 either side. _CORPUS_FLOORS lowered 70 -> 67 deliberately; the tolerated floor of 22 still stands under 29. W2's example line for '田中さん 様.' goes with the flag rather than with the clause it witnessed, which lands in the next commit: the plan held it back, but test_the_tolerated_corpus_is_disjoint_from_the_contract_ones refuses a text that is tolerated in the case table while a contract corpus still holds it -- the "demoted and still enforced" state nobody reads for. So corpus_rules.jsonl 248 -> 247 here, and the demotion is complete at this commit rather than half-made. W2 keeps the clause and loses its witness for one commit. Gate at four baselines: 352 / 247 / 155 / 14 intentional, unexplained 0, radar unclassified 0, exit 0, no MOVED SHAPE and no NOT CHECKED. compare.py's #501 block comment still calls '田中さん 様.' contract-tier and is stale from here; it is re-measured two commits on. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…lerated zone
W2 read "The split-off ignores surrounding punctuation, but never
treats a part that is not name text as the name's end". Native CJK
writing puts no punctuation around a name for a split-off to ignore, so
every input the clause describes is a listing form -- W3's ground, and a
description rather than a promise. The clause had no pure witness and
could not be given one: its example, '田中さん 様.', left in the
previous commit with the flag, and the two boundary lines that might
have looked like witnesses for the second half ('김지양', '王君') are
about the vocabulary's own limits, not about punctuation. The second
half was already stated at more length in W3 ("A part that is nothing
but suffix words is not the name's end: it is declined as the site"), so
both halves went.
W3 gains the example line, one descriptive sentence, and a rationale
clause naming the punctuation a listing carries in beside the comma
itself; its `tolerated:` marker now counts five illustrated-not-promised
lines where it counted four. W2 keeps its glued-honorific statement and
five examples, none carrying ASCII -- measured 2026-09-05, W2's line was
the only CJK example outside W3 that carried an ASCII character at all.
Both `interacts:` lines still hold: W2 points at W3, where the crossing
and now the punctuation reading live.
_script_segment.py's W2 citation is trimmed to the surviving statement
in the same commit (test_citations_are_verbatim_excerpts catches it) and
says where the punctuation half went.
corpus_rules.jsonl is unchanged at 247: build_rules_corpus.py skips a
tolerated rule's examples whole, so the line arriving in W3 adds nothing
back. decisions.md#cjk-comma-demotion takes the record -- the criterion
the 2026-09-01 sweep used, the wider one that replaces it, the
parenthesized-nickname row kept and the delimiter measurement behind
that keep, and the recompute for the sweep itself.
Gate at four baselines: summary lines byte-identical to the previous
commit's -- 352 / 247 / 155 / 14 intentional, unexplained 0, radar
unclassified 0, exit 0.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The four #501 contest rows closing the 2.0.0 watched section carried "Three are contract tier (corpus_cjk.jsonl; the first also corpus_rules.jsonl), so a move on them fails the run". Two are, now: '김민준 박사님' and '선생님'. '田中さん 様.' left both contract files in this bundle's first two commits and reads radar, so a shape move on it prints and no longer fails. That is the thing worth stating and not the digit: a row's fatality follows the tier its NAME reads at, which no roster in this file records, so a demotion elsewhere changes what a row here costs without touching the row -- the comment is the only thing that goes stale, and only a re-measurement finds it. Mutation controls, in memory around main() with the worktree untouched, 2026-09-05, and run as a PAIR because either alone is consistent with a dead check: '田中さん 様.' corrupted at 2.0.0 -- MOVED SHAPE (radar) expected_since_2.0.0.toml: 1 watched diff shape(s) disagree with this run. [...] This does not fail the run because the name is radar tier (_CORPUS_TIERS), which watches without promising. main() returns 0. It returned 1 before this bundle. '선생님' corrupted the same way -- MOVED SHAPE expected_since_2.0.0.toml: [...] This fails the run because the name is contract tier (_CORPUS_TIERS). main() returns 1, so the exit 0 above is the tier and not a check that stopped running. _CORPUS_TIERS's "no text in the tolerated file is held by a contract corpus" sentence is re-verified and gains what the demotion taught it: '田中さん 様.' was one of the two PURE texts the 2026-09-01 W2 swap brought in, so marking its case row was not enough on its own -- the example line had to move into the tolerated W3 in the same breath, which test_the_tolerated_corpus_is_disjoint_from_the_contract_ones enforces on every run. decisions.md's watched-shapes measurement, taken hours before the demotion, is corrected in the same commit for the same claim. tools/differential/README.md needed no edit: its "three contract-tier names" sentence is about the #498 rule-pair swap, a different three. Full suite 6507 passed, ruff clean, mypy clean over 110 files. Gate at four baselines: 352 / 247 / 155 / 14 intentional, unexplained 0, radar unclassified 0, exit 0, no MOVED SHAPE and no NOT CHECKED. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…nored, it is decisive
W2's clause read "The split-off ignores surrounding punctuation, but
never treats a part that is not name text as the name's end". The
previous commit moved BOTH halves to W3 on a rationale that reaches
only the first, and the second half is normative: it is what the
parser does with no comma and no period anywhere in the input, and
contract-tier pure names depend on it.
1. W2's SECOND HALF RESTORED, with pure witnesses. Measured 2026-09-05
by forcing _is_post_nominal to False in memory -- the predicate the
site scan _peel_site reads -- and re-parsing (family, given, suffix,
title): '田中さん 様' ships ('田中', '', 'さん, 様', '') and gives
('田中さん', '', '様', '') without the step; '김민준 박사님' ships
('김', '민준', '박사님', '') and gives ('김', '민준', '박사, 님', '');
'선생님' ships ('선생님', '', '', '') and gives ('선', '생', '님', '').
The post-nominal word is dissected in every case. '김민준 박사님' and
'선생님' are corpus_cjk.jsonl names on the CONTRACT tier, and
ko_honorific_glued_doctor_spaced is the #308/#271 regression row for
the first -- under W2's remaining statement alone, 박사님 ends in the
listed tail 님 and reads as peelable. The sentence goes back in W2's
register and takes two witnesses whose texts corpus_cjk.jsonl already
held: '김민준 박사님' -> suffix="박사님" and the boundary line '선생님'
-> family="선생님". '田中さん 様' would have been the third and is NOT
added -- it is not a case row, and a new contract name needs a 1.4.0
classification this bundle does not have.
2. W3's NEW DESCRIPTIVE SENTENCE REWRITTEN to what is measured. It
attributed the crossing to the punctuation; the punctuation does not
license it and is not ignored either. W3 now says the step past a
post-nominal word is W2's and fires with no punctuation present, that
a period on a SEPARATE post-nominal word rides into the suffix and
moves no division, and that a period glued to the honorific's own word
decides against the split-off. The '田中さん 様.' example line stays;
'田中さん.' is not a case-row text, so it is described in prose rather
than given an example line.
3. decisions.md#cjk-comma-demotion. The moved-clause note is rewritten
as a within-branch correction: the FIRST half had no pure witness and
moved to W3 as description, the SECOND is normative, pure-witnessed
and was cut in error. "Each the spaced twin of a comma row that WAS
demoted" is corrected -- '田中さん, 様.' and '김민준, 씨.' are rows,
'김민준, 양.' is not; enumerating the CASES texts containing 양 gives
김지양, 양 미선, 김민준 양, 김민준 양. and 양 지훈, and '김민준 양.'
is demoted on the class argument alone. A FOUND bullet records the
period measurement: '田中さん 様' -> family 田中, suffix 'さん, 様';
'田中さん 様.' -> family 田中, suffix 'さん, 様.'; '田中さん 太郎.' ->
family '太郎.', given '田中さん' (family '田中さん', given '太郎'
without the period -- the order flips); '田中さん.' -> title
'田中さん.'; '김민준씨.' -> title '김민준씨.'. The one input where a
period changes nothing is the one where it sits on a word the
split-off steps past regardless.
4. The watched-shapes quiet-tree figure is PINNED to the commit it was
measured at. This branch's 214a1ed touched _script_segment.py, so
a965272..HEAD is 6 / 41 / 16 where the paragraph claims 5 / 38 / 15 --
which is a965272..4dd776a, still true there. Recorded as a point
measurement with the next touching commit named.
5. Two stale enumerations gain the third demoted class, matching
cases.py's wording: compare.py's _CORPUS_TIERS comment and
build_cjk_corpus.py's module docstring both listed only the comma
listing and the Latin title/credential.
6. The three period rows' notes drop the repeated "the 2026-09-01
sweep's 'comma or ASCII letter' criterion could not see" -- provenance
about a past oversight, stated three times where the decisions note
states it once. The constraint half and the #320-at-HEAD half stay.
_script_segment.py carries the restored sentence as a second W2
citation, on _peel_site where the scan-back lives, and the
_peel_honorific_tail block is corrected to what W3 now says.
corpus_rules.jsonl 247 -> 249, both arrivals already in
corpus_cjk.jsonl, so the compared pool is unchanged at 1113 / 1120.
Full suite 6509 passed, 0 xpass; ruff clean; mypy clean over 110
files. Gate at four baselines: 352 / 247 / 155 / 14 intentional,
unexplained 0, radar unclassified 0, exit 0, no MOVED SHAPE and no NOT
CHECKED. All four generated corpora regenerate byte-identical.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…, and W3 says what pins it Five prose corrections to what this bundle's own review round wrote, each re-measured on this tree at d0faed4. The recorded-shapes snapshot contrasted its pinned quiet-tree figure with a live one "at HEAD", and that live figure was already stale at the commit that wrote it -- the exact defect the pin exists to avoid. The contrast is removed rather than refreshed: the pinned 5 / 38 / 15 over a965272..4dd776a is now the only number, and the paragraph says instead that 214a1ed and d0faed4 touched _script_segment.py's comments and W2/W3 citations, so a HEAD recompute is larger and moves with every further comment-only commit. The RECOMPUTE line now insists on the pinned endpoint. The W2 bullet's summary of its three forced parses said "in every case the post-nominal word is dissected". Re-run with _is_post_nominal rebound to False, only two are: 박사님 -> 박사, 님 and 선생님 -> 선, 생, 님. The first, 田中さん 様, yields ('田中さん', '', '様', '') -- 様 whole, but taken as the split site, so it breaks the other clause, never the name's end. The bullet now names which clause each case witnesses. The same bullet had the polarity of its own argument backwards, saying a W2 reduced to its first sentence "would have stopped contradicting" the regression its example pins. parse('김민준 박사님').suffix is '박사님'; under the first sentence alone 박사님 ends in the listed tail 님 and reads as peelable, so the reduced W2 would have BEGUN contradicting ko_honorific_glued_doctor_spaced -- which is the argument for restoring the half, not against it. The FOUND bullet said "four parses" over a list its own later sentences call "the five inputs": four counts the sentences, one of which carries two strings. It is five now, by input, with the convention stated and the sixth string (the no-period twin 田中さん 太郎) accounted for. W3's new period sentence asserted that 田中さん. and 김민준씨. each read as a title, and neither string is a case row or in any corpus, so nothing pins those readings. W3 is tolerated and illustrates rather than promises, and a new name would need a baseline classification this bundle does not carry, so no rows are added; the sentence says instead that it reports parses measured 2026-09-05 that no row pins and that can move unannounced, pointing at decisions.md#cjk-comma-demotion. _script_segment.py's comment restating the same reading is aligned to say the same. Verified: 698 passed on the doc/rules/decisions/mechanisms/guard/citation subset, ruff clean, and the differential at --baseline 2.2.0 exits 0 with 14 intentional diffs, 0 unexplained, 0 radar unclassified. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #505 +/- ##
=======================================
Coverage 98.66% 98.66%
=======================================
Files 45 45
Lines 3219 3219
=======================================
Hits 3176 3176
Misses 43 43 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three CJK case rows whose text ends in an ASCII period after an honorific —
田中さん 様.,김민준 씨.,김민준 양.— declaretoleratedand leave the contract tier, closing a gap in the 2026-09-01 demotion's sweep criterion. Refs #469 (the demotion this completes) and #501 (one of its four contests changes severity).What was wrong
The 2026-09-01 sweep demoted every CJK-bearing row carrying a comma or an ASCII letter. A trailing ASCII period is neither, so three
fix(#320)probe rows — the spaced twins of comma rows that were demoted — stayed contract, and one of them was then chosen as W2's replacement example because it passed the same test. A period after a CJK honorific is a listing artifact no writing system produces: exactly the class the doctrine demotes. And W2's clause "the split-off ignores surrounding punctuation" had no pure witness, because native writing has no punctuation around a name to ignore — which makes it descriptive, not a promise.Measured 2026-09-05 with the wider criterion (any non-space ASCII in a CJK-bearing untolerated row): eight texts. The three period rows are this PR.
山田 太郎 (マイケル・ジャクソン)is kept: the parser reads ASCII, fullwidth and corner-bracket delimiters identically, and what the row pins is the nakaguro inside a delimited nickname. The four mixed-script rows (John 王,王·Smith,Andersonさん,Anderson선생님) were reviewed and kept byd562164, for reasons that still hold.What changes
tolerated=Truewith the reason in their notes; they still pin the A trailing period stops a CJK honorific being recognized ("김민준, 씨." → given "씨.") #320 mechanism at HEAD. Shape 6/7 purity now refuses any ASCII character other than a space (letters and commas already were; punctuation and digits join them); untagged rows are untouched, so the mixed-script rows stay whered562164put them.corpus_cjk.jsonl73 → 70,corpus_cjk_tolerated.jsonl26 → 29, the contract floor lowered deliberately.田中さん 様.example. The other half — a part that is not name text is never the name's end, so the split-off steps past a post-nominal word and never dissects it — is normative and stays in W2, now witnessed by pure contract-tier names (김민준 박사님→ suffix박사님;선생님as the boundary): forcing that step off in memory dissects박사님into박사, 님and선생님into three, the Glued CJK honorifics are absorbed into the name (山田太郎様 → family 山田太郎様) #308/Unspaced Chinese/Korean names: surnames constants + longest-match segmentation #271 regression the case rows pin. The branch's first draft took both halves; the design-docs review caught it, anddecisions.mdrecords the correction.corpus_rules.jsonl248 → 247 → 249, every arrival already incorpus_cjk.jsonl. The example line moved in the first commit, not the second:test_the_tolerated_corpus_is_disjoint_from_the_contract_onesrefuses a text that is demoted and still enforced, so no half-demoted state exists on the branch.田中さん 様.divides where田中さん 様does), while a period glued to the honorific's own word is read and decides against the split-off —田中さん.and김민준씨.each parse as a title — and田中さん 太郎.versus田中さん 太郎flips the order. Five parses in the decisions entry.decisions.md#cjk-comma-demotiongains a dated supersession note (the criterion was narrower than the doctrine it implemented; the widened rule; the nickname keep with its delimiter measurement; the clause move). The_WATCHED_DIFFSSix contested diffs in the 2.x ledgers have no pin, and the empty rosters read as if none existed (MD, PHDgoes to whichever rule comes first) #501 block's "three contract-tier" becomes two, and the watched-shapes arc's measurement paragraph is corrected to match.Verification
Intentional counts 352 / 247 / 155 / 14 unmoved at every commit (a demotion moves nothing, and the two W2 witnesses were already pooled), 0 unexplained, 0 radar-unclassified, no finding block. Mutation control:
田中さん 様.'s watched row corrupted now printsMOVED SHAPE (radar)and exits 0 where master exited 1, and선생님corrupted identically still exits 1 — the severity followed the tier, and the check is live. Full suite green; ruff and mypy clean. Re-running the sweep after: 8 → 5 texts, the period class gone, the letter class untouched.🤖 Generated with Claude Code